What does this PR do?

This PR adds a test that ensures Beatsreceivers ingesting logs can:

• Start running with a Log input configuration and ingest logs
• Stop, then re-start with the feature flag to run Filestream input instead of the Log input
• The Filestream input running from the Log input configuration continues ingesting the file from where the Log input had stopped.

Currently there is no way to run a Beatreceiver directly from a standalone Beats binary, hence the test in this repo.

Why is it important?

It allows us to run the Filestream input instead of the Log input for a Log input configuration

Checklist

• I have read and understood the pull request guidelines of this project.
• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Author's Checklist

• Wait for [Filebeat] Filestream running as Log input under Elastic Agent or feature flag beats#46587 to be merged
• Remove overwrites
  • dev-tools/notice/overrides.json
  • go.mod
• Update comments on testing/integration/ess/logfile.go
• Replace testing/integration/ess/logfile.go with the functionality provided by Add TempDir and NewLogFile test utilities elastic-agent-libs#369

## Disruptive User Impact

How to test this PR locally

Running integration tests

Run the test:

1. Package the Elastic Agent for your OS

   DEV=true SNAPSHOT=true EXTERNAL=true PACKAGES="tar.gz" PLATFORMS=linux/amd64 mage -v package
   

2. Run TestFilebeatReceiverLogAsFilestream from testing/integration/ess/otel_log_as_filestream_test.go

   AGENT_VERSION="9.4.0-SNAPSHOT" SNAPSHOT=true TEST_PLATFORMS="linux/amd64" mage -v integration:single TestFilebeatReceiverLogAsFilestream
   

Manual test

Run the Elastic Agent in Otel mode setting features.log_input_run_as_filestream.enabled: true for the filebeatreceiver you want to test.

1. Package the Elastic Agent for your OS

   DEV=true SNAPSHOT=true EXTERNAL=true PACKAGES="tar.gz" PLATFORMS=linux/amd64 mage -v package
   

2. Create a log file

   docker run --rm mingrammer/flog -n 50 > /tmp/flog.log
   

3. Extract and run Elastic Agent in otel mode using a configuration like the one below

   ./elastic-agent otel --config otel.yml
   

   otel.yml

   receivers:
     filebeatreceiver:
       filebeat:
         inputs:
           - type: log
             id: a-unique-id
             allow_deprecated_use: true
             paths:
               - /tmp/flog.log
       output:
         otelconsumer:
       logging:
         level: debug
         selectors:
           - '*'
       features.log_input_run_as_filestream.enabled: true
   
   exporters:
     elasticsearch:
       user: elastic
       password: changeme
       endpoints:
         - https://localhost:9200
       tls:
         insecure_skip_verify: true
       logs_index: my-index
       sending_queue:
         enabled: true
         wait_for_result: true # Avoid losing data on shutdown
         block_on_overflow: true
       mapping:
         mode: none
   
   service:
     pipelines:
       logs:
         receivers: [filebeatreceiver]
         exporters: [elasticsearch]
     telemetry:
       logs:
         level: DEBUG
         encoding: json
         disable_stacktrace: true

Related issues

• Blocked by [Filebeat] Filestream running as Log input under Elastic Agent or feature flag beats#46587

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...